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Introduction 

It is over the ten years since lEA last investigated civic education, and in that time 
new challenges have emerged in educating young people for their roles as citizens in 
the 2E' century. These challenges have stimulated renewed reflection on the 
meanings of citizenship and the roles of and approaches to civic and citizenship 
education. In many countries there is a growing interest in using evidence to improve 
policy and practice in civic and citizenship education. 

The purpose of the International Civic and Citizenship Education Study (ICCS) is to 
investigate the ways in which young people are prepared to undertake their roles as 
citizens in a range of countries. The study will report on student achievement on a test 
of conceptual understandings and competencies in civic and citizenship education. It 
will also collect and analyse data about student dispositions and attitudes relating to 
civic and citizenship education. The study builds on the previous lEA study of civic 
education (CIVED) undertaken in 1999. A website with information about ICCS can 
be found at http://iccs.acer.edu.au/ . 

It is recognised that there is substantial diversity in the field of civic and citizenship 
education within and across countries. Consequently, maximising the involvement of 
researchers from participating countries in this international comparative study is 
deemed to be of particular importance for the success of this study. Input from 
national research centres will be sought throughout the study and the consortium will 
develop strategies to encourage country contributions to instrument development as 
well as to the dissemination of results. 

This document will give an overview of the study and summarise the different aspects 
of its implementation. It also outlines the structure of the assessment framework and 
the process of the development and implementation of instruments as well as the 
benefits for participating countries. 

Research Questions and Study Coordination 

The purpose of the International Civic and Citizenship Education Study (ICCS) is to 
investigate the ways in which young people are prepared to undertake their roles as 
citizens in a range of countries in the 2E' century. In pursuit of this purpose, the 
study will report on student achievement in a test of knowledge, conceptual 
understandings and competencies in civic and citizenship education. It will also 
collect and analyse data about student attitudes to civics and citizenship and 
participation in civic activities. The study builds on the previous lEA studies of civic 
education, particularly the CIVED study in 1999. Crucially, it acknowledges the need 
for a new study and is a direct response to the challenges of educating young people 
across the world in changed contexts of democracy and participation. This purpose is 
reflected in the key research questions for the study. 

The key research questions for the study concern student achievement, dispositions to 
engage, participation in civic-related activities and attitudes related to civic and 
citizenship education. Specifically the key research questions are as follows. 

RQ 1 What variations exist between countries, and within countries, in student 
achievement in conceptual understandings and competencies in Civic and 



Citizenship? Analysis to address this research question would focus on the 
distribution of student achievement based on test data. 

RQ2 What changes in civic knowledge and engagement have occurred since the 
last international assessment in 1999 and what is the variation in those 
changes? This research question is mainly concerned with analysing trends 
from CIVED to ICCS and would be limited to data from countries 
participating in both assessments. 

RQ3 What is the extent of interest and disposition to engage in public and political 
life among adolescents and which factors within or across countries are 
related to it? This research question will address the issue of apathy with 
indicators of civic engagement being compared within and across countries 
and related to explanatory variables at various levels. 

RQ4 What are adolescents’ perceptions of the impact of recent threats to civil 
society and responses to these threats on its future development? The analysis 
would be primarily based on student comprehensions of the relationship 
between securing societies and safeguarding civil liberties and student 
attitudes towards citizenship rights. 

RQ5 What aspects of schools and education systems are related to achievement in 
and attitudes to Civic and Citizenship including: 

(a) General approach to civic and citizenship education, curriculum or 
program content structure and delivery. The analysis requires additional data 
to be collected at the national level data on curriculum and programmes as 
well as reports from school and teacher questionnaires. 

(b) Teaching practices such as those that encourage higher order thinking and 
analysis in relation to civic and citizenship. The analysis would be based on 
data about student perceptions of and teacher reports on instructional 
practices. 

(c) Aspects of school organisation including opportunities to contribute to 
conflict resolution, participate in governance processes, and being involved in 
decision making. The analysis requires data on student perceptions of school 
governance and reports from school principals or civic education head 
teachers (where appropriate). 

RQ6 What aspects of student personal and social background, such as sex, 
socioeconomic background, language background, are related to student 
achievement in and attitudes towards Civic and Citizenship education? The 
analysis would be based on data from the student background questionnaire, 
the attitude questionnaire and the assessment of knowledge, understanding and 
competencies. 

It is important to recognise that ihe International Civic and Citizenship Education 
Study (ICCS) is a collaborative project involving groups of researchers and individual 
scholars around the world. Staff from research centres in participating countries, the 
consortium, the lEA and its institutions, expert consultants and funding agencies will 
work together on this study. 




Table 1 shows those educational systems which are currently participating or have 
expressed interest in the International Civic and Citizenship Education Study. More 
countries or educational systems may still join prior to the field trial in the second half 
of 2007. 

Table 1 Countries that are currently participating or have expressed in interest 
in participation 



Australia 


• France 


• 


Northern Ireland 


Austria 


• Germany 


• 


Norway 


Brazil 


• Greece 


• 


Paraguay 


Canada 


• Guatemala 


• 


Poland 


Chile 


• Hong Kong SAR, China 


• 


Portugal 


Chinese Taipei 


• Indonesia 


• 


Russian Federation 


Colombia 


• Italy 


• 


Scotland 


Cypms 


• Latvia 


• 


Slovenia 


Denmark 


• Lithuania 


• 


Spain 


Dominican Republic 


• Luxembourg 


• 


Sweden 


EnglandAVales 


• Mexico 


• 


Switzerland 


Estonia 


• Netherlands 






Finland 


• New Zealand 







National Research Coordinators (NRC) play a major role in the lEA projects and 
coordinate the work at the national centres responsible for implementing the study in 
each participating country. In addition to the national coordinator, additional staff 
members are required in to undertake the study at the national level including data 
managers, data entry staff, experts for instrument translation, test administrators and 
other office staff. The national coordinator is the main contact in each country for the 
international study coordination and is responsible for coordinating all tasks within 
the country. 

It is envisaged that regional modules will be implemented for groups of participating 
countries from the same region. Currently, two regional modules for Europe and Eatin 
America are being established. Eor each regional module, region- specific instrument 
components are being developed, which will be administered after the international 
items material. 

The study is organized around a consortium of three partner institutions, which 
cooperates with the lEA Secretariat, the lEA Data Processing Center (DPC) and the 
national research coordinators (NRCs): The Australian Council for Educational 
Research (ACER), the National Eoundation for Educational Research (NEER) in the 
United Kingdom and the Laboratorio di Pedagogia sperimentale (EPS) at the Roma 
Tre University. 

The consortium combines expertise in the field of civic and citizenship education, 
experience in internationally comparative large-scale assessments and broad research 
capacities and key staff members from ACER, NEER and EPS have expertise in civic 
and citizenship, at national, regional and international levels, which can be combined 
with experience in test development, coordination of international as well as national 
large-scale assessments, scaling and statistical analysis. 

Communication between the partners, lEA institutions and national research is 
facilitated by a secure Internet website which can be accessed by national research 




centres, international consortium partners and lEA institutions. Relevant study 
documentation (meeting documents, manual, instruments, contact addresses etc.) are 
posted on this website. 

The international study will be organised by the following organisations: 

• The International Study Centre (ISC) has been be established at the Australian 
Council for Educational Research (ACER) in Melbourne/ Australia. Apart 
from providing general technical direction and coordination for the study, the 
International Study Centre will be responsible for the International Test 
development, the development of the Student Background Questionnaire, 
scaling procedures, analysis and reporting. 

• The National Foundation for Educational Research (NEER) in Slough/United 
Kingdom will be mainly responsible for the coordination of the development 
of the National Contexts Survey and the Student Perceptions Questionnaire as 
well as the coordination, development, and reporting of a European Regional 
Component. 

• The Laboratorio di Pedagogia sperimentale (EPS) at the Roma Tre University 
(Italy) will be mainly responsible for the development of the Teacher and 
School Questionnaire and contribute to the development of the international 
test. 

• The lEA Data Processing Center (DPC) in Hamburg (Germany) will be 
responsible for the development of field procedures, data mana^ment, 
sampling and weighting. 

• The lEA Secretariat in Amsterdam (Netherlands) will manage the 
relationships with participating countries and coordinate the translation 
verification of survey instruments and the implementation of quality 
monitoring. 

The Joint Management Committee (JMC) of the project is chaired by the Project 
Coordinator (John Ainley, ACER) and includes the research directors at the 
International Study Centre (Wolfram Schulz, ACER) and its partner institutions 
(David Kerr, NEER and Bruno Losito, EPS) as well as representatives from the lEA 
Data Processing Center and the lEA Secretariat. The committee is responsible for the 
overall management of ICCS and meets in regular video- and teleconferences on a 
monthly basis. 

The Project Advisory Committee (PAC) consists of experts in civic and citizenship 
education and international comparative research and provides advice on the 
conceptual framework, instrument development, measurement issues, data analysis 
and reporting. Members of this committee ^present different cultural backgrounds 
and educational systems bringing expertise from a variety of relevant disciplines in 
the field. 

ICCS also makes use of a number of expert consultants at appropriate points in the 
study. Expert consultants are identified based on their specialist expertise in relation 
to a number of areas of relevance to the study. 

The Assessment Framework 

The International Civic and Citizenship Education Study (ICCS) incorporates an 
Assessment Eramework. This Eramework builds upon and extends the range of the 




previous lEA Civic Education Study in 1999 (hereafter CIVED) as well as reflect 
recent developments in civic and citizenship education assessment (Torney-Purta et 
al, 1999; Torney-Purta et al, 2001; Amadeo et al, 2002). Eormal curricula and 
underlying concepts in civic and citizenship education are different a decade after the 
commencement of CIVED (Banks, 2004; Kymlicka, 2001; Menezes et al., 2004). 
Articulated learning outcomes, clarified curricula and explicit frameworks are now 
more common (Birzea et al., 2004; EURYDICE, 2005; Eee at al., 2004). 

The framework takes account of the research literature in the field, especially in 
relation to the connections between the concepts of civic and citizenship and 
accommodates the core concepts and themes or issues contained in the relevant 
curricula from those member countries which provided material for consideration 
during the proposal development in 2005 ^ . 

The Assessment Eramework comprises a Civics and Citizenship framework, which 
will guide the development of cognitive test and student perceptions questionnaire, 
and a contextual framework, which will serve as a reference point for the 
development of student background, teacher, school and national context 
questionnaires. 

Given the central role of the Assessment Eramework in the process of instrument 
development, it has been important to: 

• maintain a strong connection to the constructs used in the lEA CIVED study 
in 1999; 

• reflect contemporary research understandings of manifestations of civic and 
citizenship education in school students; 

• meet the needs of participating countries; 

• address the research questions outlined in the ICCS proposal for the lEA 
General Assembly 2005; 

• include only content that can be measured; 

• comprise content descriptors that are agreed to be significant, discrete and that 
describe the breadth of civic and citizenship education in school students; and 

• address the contexts within which civic and citizenship education takes place. 

The Assessment Eramework is divided into two parts: 

• The Civics and Citizenship Framework outlines the aspects that will be 
addressed when collecting the outcome measures through the cognitive test 
and the student perceptions questionnaire. 

• The Contextual Framework provides a mapping of context factors that may 
influence outcome variables and explain their variation. 



The countries that responded to the invitation to provide materials were Belgium, Germany, Czech 
Republic, France, Macedonia, Estonia, the Netherlands, Palestine, Scotland, and Sweden. 
Information was also provided by Australia, Chile and Hong Kong. 




Both parts of the Assessment Framework provide a conceptual underpinning for the 
international instrumentation for ICCS and they are also points of reference for 
regional assessment components. 

It is expected that many of the secure CIVED trend cognitive items and items from 
(some of) the attitude and concept CIVED scales will be included in the ICCS 
assessment instruments. The three domains of the CIVED conceptual model of civics 
and citizenship are: 

• Domain I: Democracy/Citizenship 

• Domain II: National Identity /International Relations 

• Domain III: Social Cohesion/Diversity 

The ICCS Civics and Citizenship Eramework is organised around three dimensions: a 
content dimension specifying the subject matter to be assessed within civics and 
citizenship; an affective -behavioural dimension that describes the types of student 
perceptions and activities that will be measured; and a cognitive dimension that 
describes the thinking processes to be assessed. 

The four content domains in the ICCS Civics and Citizenship Eramework are: 

• Content Domain I : Civic Society and Systems 

• Content Domain 2: Civic Principles 

• Content Domain 3: Civic Participation 

• Content Domain 4: Civic Identities 

It is important to distinguish the different types of student perceptions and behaviours 
that are relevant in the context of civics and citizenship. Eor this purpose, three 
affective- behavioural domains in the ICCS Civics and Citizenship Eramework are 
identified in this assessment framework: 

• Affective -behavioural Domain 1: Value beliefs 

• Affective -behavioural Domain 2: Attitudes 

• Affective -behavioural Domain 3: Behavioural intentions 

• Affective -behavioural Domain 4: Behaviours 

Similar to the assessment framework for TIMSS (Mullis et. ah, 2005), cognitive 
domains define the cognitive processes assessed with test items. The two cognitive 
domains in the ICCS Civics and Citizenship Eramework are: 

• Cognitive Domain I: Knowing 

• Cognitive Domain 2: Reasoning and Analysing 

The ICCS assessment of the outcomes of civic and citizenship education comprises 
three instruments: 

• A cognitive test 

• A student perceptions questionnaire 




• A section of the student background questionnaire dealing with active 
citizenship behaviours. 

The data from the cognitive test are to be used to construct a scale of civic and 
citizenship knowledge and understandings as described by the two cognitive domains, 
and representing the substance of the four content domains. 

The data from the student perceptions questbnnaire are to be used to articulate 
constructs pertaining to the four affective- behavioural domains and relating to the 
substance of the four content domains. The amount and type of assessment 
information accessed by each instrument will vary across the four content domains. 

The data from the student background questionnaire dealing with active citizenship 
behaviours are used both as indicators of active citizenship and as possible 
explanatory variables of civic and citizenship achievement. Indicators of student 
activities are also important context variables and are included in the contextual 
framework. 

Table 2 Relationship between cognitive or affective -behavioural and content 
domains 





Content domain 1; 

Civic Society and 
Systems 


Content domain 2; 

Civic principles 


Content domain 3: 

Civic Participation 


Content domain 4: 

Civic Identities 


Cognitive domains 




Knowing 


I 


II 


III 


IV 


Analysing and 
reasoning 


v 


VI 


VII 


VIII 


Affective -behavioural 
domains 




Value beliefs 


A 


B 


c 


D 


Attitudes 


E 


F 


G 


H 


Behavioural intentions 






I 




Behaviours 






J 





Table 2 shows a mapping for items that may be placed in different cells and relate to 
either cognitive or affective-behavioural domains as well as content domains. 
Cognitive items from both domains (knowing; reasoning and analysing) and affective- 
behavioural items from two domains (value beliefs and attitudes) will be developed in 
the contexts of all four content domains. These mappings will be guided by the 
compatibility of each content domain to the different affective-behavioural and 
cognitive domains and will therefore not necessarily be evenly spread across the 
content domains. Items developed to measure behavioural intentions or actual 
behaviours will only relate to content domain 3. 

A study of civic-related learning outcomes and indicators of civic engagement needs 
to be set in the context of the different factors influencing them. Young people 
develop their understandings about their roles as citizens in contemporary societies 
through a number of activities and experiences that take place within the contexts of 
home, school, classrooms, and the wider community. 




































Therefore, it is important to recognise that young people's knowledge, competencies, 
dispositions and self-beliefs are influenced by variables that can be located at different 
levels in a multi-level structure. The individual student is located within overlapping 
contexts of school and home. Both contexts form part of the local community which, 
in turn, is embedded in the wider sub -national, national and international context. For 
the contextual framework for ICCS the following levels will be distinguished: 

• Context of the wider community: This level comprises the wider context 
within which schools and home environments work. Factors may be found at 
local, regional and national levels. It should be noted that for some countries 
the supra-national level might also be of relevance, as for example in member 
countries of the European Union. 

• Context of schools and classrooms: This level comprises factors related to the 
instruction students receive, the school culture and the general school 
environment.^ 

• Context of home environments: This level comprises factors related to the 
home background and the social out-of- school environment of the student (for 
example, peer-group activities). 

• Context of the individual: This level includes individual characteristics of the 
student. 

Another important distinction can be made by grouping contextual factors in those 
related to either antecedents or processes: 

• Antecedents are those factors that affect how student learning and acquisition 
of civic-related understandings and perceptions takes place. It should be noted 
that these factors are level- specific and may be influenced by antecedents or 
processes at a higher level (for example, civic-related training of teachers may 
be affected by historical factors and/or policies implemented at the national 
level). 

• Processes are those factors related to the civic-related learning and the 
acquisition of understandings, competencies and dispositions. They are 
constrained by antecedents and also influenced by factors on higher levels of 
the multi-level structure. 

Antecedents and processes are factors that shape the outcomes at the level of the 
individual student. It should be noted that learning outcomes related of civics and 
citizenship education at the student level can also be viewed as aggregates at higher 
levels (school, country) where they can have an effect on factors related to process 
(for example, higher levels of civic understandings and engagement among students 
can influence the way civic and citizenship education is taught at school). 



^ It is important to note that given the sampling design for ICCS school level and classroom level 
cannot be disentangled as (generally) only one classroom will be selected within each sampled school. 




Figure 1 Contexts for the Development of Learning Outcomes related to Civics 
and Citizenship 



Antecedents Processes Outcomes 




Figure 1 illustrates which contextual factors may influence learning outcomes of civic 
and citizenship education. It should be noted that the (double- headed) arrow between 
processes and outcomes signals that there is a reciprocal relationship: It is important 
to emphasise that the there is a "feedback" between civic-related learning outcomes 
and processes; for example, students with higher levels of civic knowledge and 
engagement are also more likely to participate in activities (at school, at home and 
within the community) promoting these outcomes. 

The (single- headed) arrow between antecedents and processes describes the 
relationship between these two types of factors at each level as uni- directional. 
However, higher- level processes can influence antecedents and it is also likely that in 
a long-term perspective outcomes may affect variables that are antecedents for 
learning processes. 

Within this general contextual framework for ICCS, it is now possible to map 
variables for which data are collected to a three- by- four grid with antecedents, 
processes and outcomes as columns and the levels of nation/community, 
school/classroom, and home environment as rows. The last column for outcomes is 
not split into levels though it should be recognised that for analysis purposes 
aggregated data may also be used at country- and school/classroom levels. 
















Table 3 Mapping of variables to contextual framework (examples) 



Level of... 


Antecedents 


Processes 


Outcomes 


National and other 
communities 


NCQ & other sources: 

Democratic history 
Structure of education 


NCQ & other sources: 

Intended curriculum 
Political developments 


StT & StPQ & StBQ: 

Test results 
Student perceptions 
Student behaviours 


School/classroom 


ScQ & TQ: 

School characteristics 
Resources 


ScQ & TQ: 

Implemented curriculum 
Policies and practices 


Student 


StBQ: 

Gender 

Age 

Language 


StBQ: 

Learning activities 
Practised engagement 


Home environment 


StBQ: 

Parent SES 
Ethnicity 


StBQ: 

Communication 
Peer-group activities 



NCQ: National Context Survey; ScQ: School Questionnaire; TQ: Teacher Questionnaire; StBQ: 
Student Background Questionnaire; StPQ: Student Perceptions Questionnaire; StT: Student Test; 
SES : Socio-Economic Status 



Table 3 provides a mapping of examples of potential variables (or groups of 
variables) collected with different ICCS instruments to each cell in this grid. 
Variables related to the context of the nation/community would be primarily collected 
through the National Context Survey and other possible data sources. Variables 
related to the context of schools and classrooms would be collected through school 
and teacher questionnaires. The Student Background Questionnaire provides 
information on antecedents of the individual student and the home environment as 
well as about some process-related variables (for example: learning activities). 
Student Test and Student Perceptions Questionnaire would collect data on outcomes. 
In addition, the Student Background Questionnaire will include questions regarding 
student participation in civic-related activities, which will also be used as indicators of 
active citizenship related to Domain 3 (Civic Practices). 

It should be noted that there are some potential variables that may be measured at one 
level pertaining to another level and that are not included in the mapping in Table 3 : 
Student observations of learning practices in the classroom may be aggregated and 
used as classroom or school variables. Furthermore, student, school and teacher 
questionnaires may also provide civic-related information about the context of the 
local community. 

Instrument Development 

The following instruments will be developed for ICCS: 

• International cognitive test (60 minutes): This instrument will contain of a 
rotated booklet design with up to 80 test items (including multiple -choice and 
open-ended response item types). 

• International student background questionnaire (15 minutes): This instrument 
will collect information on student characteristics, family background, 
classroom practices and civic-related activities. 

• International student perceptions questionnaire (30 minutes): This instrument 
will be designed to measure students' beliefs and perceptions related to civic 
and citizenship education. 























• Regional cognitive test component (10-30 minutes): There may be region- 
specific test components which would consist of up to 20 multiple -choice 
items measuring student knowledge related to region- specific issues. 

• Regional student questionnaire (10-20 minutes): This questionnaire will be 
specifically designed for particular regional modules and will only be 
administered in countries participating in a regional module. It will contain 
student background and perceptions questions related civic and citizenship 
relevant for the regional module. 

• Teacher questionnaire (30 minutes): This questionnaire will be administered 
to teachers and ask about their perception of civic and citizenship education at 
school. 

• School questionnaire (30 minutes): This questionnaire will be administered to 
school principals and capture school- level variables related civic and 
citizenship participation. 

• National Contexts Survey: This questionnaire will be administered on-line to 
National Research Coordinators and be designed to collect data on the 
background for civic and citizenship education in each participating country. 

The primary aims of the instrument development process are to create instruments 
which: 

• unambiguously map the entire agreed assessment framework; 

• are of the highest technical and psychometric quality; and 

• provide data that address the key research questions outlined in the ICCS 
proposal endorsed by the lEA General Assembly in 2005. 

Underpinning the process of development of all instruments and the assessment 
framework in this study is the recognition of the need to facilitate contributions from 
multiple stakeholders and the iterative review process. The process is also designed to 
accommodate those variations, which will be deemed necessary by the stakeholders 
according to the particularities of the individual instruments. The development 
process involves members of the International Study Centre, its partner institutions, 
the Project Advisory Committee, and (where appropriate) expert consultants in 
partnership with national research centres and other parties involved. 

Pilot study 

The pilot study will be conducted during March and April 2007 in a subset of 
participating countries. The purpose of the pilot study is primarily to gain preliminary 
qualitative and quantitative information about the quality of the cognitive test and the 
student perceptions questionnaire. These data will be used to further evaluate and 
refine the instruments. 

Field trial 

The field trial will be conducted in all participating countries between October and 
November 2007. The purpose of the field trial is to collect sufficient quantitative data 
to make accurate judgements about the substantive and psychometric properties of the 
instruments and to evaluate and later refine the administration and data collection 
processes. 




Main study instrument finalisation 

The results of the post- field trial review will be submitted to a meeting of NRCs for 
discussion before submission to the PAC and the JMC. 

All stakeholders - consortium staff, members of the JMC, the PAC and project expert 
consultants - will have an integral role in the final instrument selection process. 

Target Population, Sampling and Weighting 

Sampling activities will be coordinated by the lEA Eata Processing Centre (lEA 
DPC) in cooperation with the International Study Centre (ISC). Jean Dumais from 
Statistics Canada has been appointed sampling referee for ICCS. He gives advice on 
sampling methodology, makes binding decisions in case of any conflicts, reviews and 
approves national sampling plans, adjudicates the sampling implementation and 
decides on whether countries will have met the sampling requirements in the main 
data collection. 

In order to reduce the burden on national centres and to ensure sampling accuracy, 
lEA DPC will undertake the drawing of all national school samples. Manuals 
describing sampling procedures and appropriate software to facilitate within-school 
sampling will be distributed to national centres. 

The population definition of students surveyed in the ICCS follows the definitions as 
used in TIMSS (2007) and includes all students enrolled in the grade that represents 
eight years of schooling, counting from the first year of ISCED Eevel 1, provided the 
mean age at the time of testing is at least 13.5 years. According to this definition, for 
most countries the target grade would be the eighth grade, or its national equivalent. 
In order to enable a link to CIVED, participating countries, which tested a different 
grade in the previous lEA survey in 1999, are encouraged to assess the same grade in 
addition to the internationally defined target grade. 

In some countries, national centres might want to restrict the coverage (for example 
by including only students taught in the dominant language in a country or by 
excluding smaller regions). In these cases, the national desired target population 
might be only a larger subset of the international desired target population described 
above. Decisions about changes in the population definition of a country have to be 
discussed with the lEA DPC and the International Study Centre. 

Eor the first pilot studies during the initial phase of instrument development, smaller 
convenience samples a used in a subset of participating countries; institutions 
involved in the instrument development will pilot material in their respective 
countries and an additional number of national centres will do further piloting. 

The international field trial material should be administered to convenience samples 
of 600-1000 students in all participating countries. The field trial samples will cover 
different school types and study programmes in each country and within-school 
sampling will be applied as part of trialling of administrative procedures. 

The sample sizes for the main data collection would be determined according to the 
sampling efficiency information from CIVED or from national surveys in the domain 
of citizenship education in order to achieve an effective sample size- of 400 students 




per country. A minimum sample size of 150 schools should be drawn for each 
country using PPS (probability proportional by size) sampling procedures and student 
samples sizes will range from 3500 to 5000 students. 

Within schools intact classrooms in the target grade will be selected. Sampling of 
intact classrooms is seen as the more appropriate approach to within- school sampling 
as it reduces the administrative burden on schools and allows questions related to 
instructional practices to be unequivocally linked to the instructional context. 

Within- school exclusions should be similar to the ones established for the TIMSS 
study: (a) Intellectually disabled students, (b) functionally disabled students and (c) 
non-native language speakers. Exclusion categories would need to be adapted to the 
national context in cooperation with the sampling referee, the lEA DPC and the 
International Study Centre. 

In many countries where an additional upper grade needs to be sampled for the 
purpose of trend comparisons with CIVED, students from this grade could be sampled 
from the same set of sampled schools. However, in other countries it may be the case 
that the upper grade is taught at a different school type and that an additional sample 
of schools could be required using the same approach to determine sample size as 
outlined above. 

In accordance with the lEA standards, sampling requirements should be established 
and sampling outcomes will be monitored to ensure that data from participating 
countries meet these requirements. It is envisaged that countries will need to have 
response school participation rates of 85 percent (before replacement) as well as 
student- level participation rates of 85 percent. The overall participation rate should 
not be lower than 75 percent. 

The ICCS teacher survey will include all teachers teaching regular school subjects to 
the students in the target grade (typically grade) at each sampled school. It will only 
include teachers who teach in grade 8 during the testing period and have been 
employed at school since the beginning of the school year. Teachers are defined as 
school staff members who provide student instruction through the delivery of lessons 
to students. Teachers may work with students as a whole class in a classroom, in 
small groups in resource rooms or one-to-one inside or outside of classrooms. 

It should be noted that there will be no link between teacher information and 
individual students and that both teachers from civic-related and not civic-related 
subjects will be surveyed. 13. In each sampled school, a minimum of 20 teachers will 
be included in the survey. It is recognised that such a relatively large sample size 
order will cause a census surveys in many schools. 

Field Operations and Quality Assurance 

As in other international studies, it will be necessary to translate the hstruments 
(international test, student background questionnaire, student perceptions 
questionnaire, teacher questionnaire, school questionnaire, regional instrument 



The effective sample size for a given multi-stage sample is equal to the size of the simple random 
sample with the same level of sampling accuracy as the multi-stage sample. The ratio between 
effective sample size and the total sample size of the multi-stage sample is called the design effect 
(see Ross, 1997). 
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components) into different languages. In addition, many questions will have to be 
adapted to the national context of a country. 

A Translation Procedures Manual (TPM) will be elaborated which will detail the 
guidelines for translation. It is envisaged that the translation will be carried out within 
each participating country by two independent translators, who should be language 
experts with appropriate knowledge in education and preferably with expertise in the 
area of civic and citizenship education. Translators would be identified and contracted 
by the national centre. The final version would be achieved by reconciling the two 
versions with the help of a third language expert in case of disagreements. 

Translation verifications of national instruments should be carried out both prior to 
field trial and main study data collections. The translation verification of instruments 
would be coordinated by the lEA Secretariat. Language experts (independent of the 
national centres) will be contracted to carry out rigorous checks of linguistic 
equivalence for each translated instrument. Results of the verification will be 
discussed between the lEA Secretariat and national centres in order to improve 
translation. Einal instruments should be submitted for a final layout verification 
coordinated by the International Study Centre. 

In view of the diverse nature of civic and citizenship education and educational 
systems across participating countries, the instruments used in ICCS will be 
characterised by substantial national adaptations. In order to ensure comparability it is 
deemed important to review and discuss national adaptations prior to the translation of 
instruments. National Adaptations Forms (NAE) will be prepared which include all 
questions in the instruments highlighting those questions which require national 
adaptation (for example questions on parental education). Decisions about the range 
of acceptable adaptations for each instrument would be made in close cooperation 
between the International Study Centre and its partner institutions . 

The field operations for the data collection in ICCS will need to be guided by 
standardised procedures that ensure the comparability of data across participating 
countries. The lEA Data Processing Center (DEC) in cooperation with the 
International Study Centre (ISC) will elaborate manuals that detail the procedures for 
different aspects of the data collection process. The following documents will be 
distributed among national centres prior to data collection (both for field trial and 
main study): 

• Sampling Procedures Manual (SMP): This document describes procedures for 
obtaining a school sample including procedures for drawing within- school 
samples of teachers and students. 

• Field Operations Manual (FOM): This document will contain a 
comprehensive description of procedural aspects of ICCS in a country from 
the delivery of instruments through to cleaning and delivery of data sets. 

• School Coordinator Manual (SCM): This document will describe the activities 
to be undertaken by the school coordinator for organising test administration 
and distribution of the Teacher and School Questionnaire within schools. It 
will have to be translated in the national language. 

• Test Administrator Manual (TAM): This document details the survey 
procedures from the beginning of test administration to the returning of 
materials to the national centre. It will have to be translated in the national 
language. 




• Marking Procedures Manual (MPM): This document will specify guidelines 
for the marking of possible open-ended responses in the ICCS test. It will have 
to be translated in the mtional language. 

• Data Entry Manual (DEM): This document provides necessary description of 
all activities related to data entry and data verification at the national centres. 
It will also contain variable description and file formats, instruction for data 
verification using the data entry software provided by the DPC and assist 
national centres with subsequent analysis of national data sets. 

Manuals would usually be accompanied by forms to be completed at national centres 
or schools, which will assist with sampling procedures, data collection and data 
verification. 

It is recognised that high levels of school participation within participating countries 
are crucial for the success of the study. Therefore, it is important to assist national 
centres with advice on how to prepare the field trial, contact schools and increase the 
willingness of schools to participate in such a study. Lessons from previous national 
or international large-scale assessments and experience with the field trial will help to 
inform about problems with refusals and ways of addressing this issue. This could be 
addressed at NRC meetings through the exchange of experiences and plenary of 
small- group discussions. 

Data collection at sampled schools will need to be undertaken by staff recruited by 
national centres within participating countries. NRCs may use external test 
administrators as well as teachers at the sampled school. In some countries, national 
centres may prefer to have school coordinators who also act as test administrators. In 
this case a combined “school coordinator and test administrator” manual would be 
provided for these countries. 

Teacher and School Questionnaires need to be administered through the school 
coordinator and appropriate procedures will be implemented to ensure high response 
rates. Consideration will also be given to the feasibility of an on-line administration of 
these questionnaires based on the experiences in other lEA studies (SITES, TIMSS). 
However, it is recognised that this approach will only be feasible in a sub-group of 
participating countries. 

Standards for lEA studies require that certain quality control measures be 
implemented during the process of data collection. The following activities described 
in other sections would all form part of quality assurance: 

• Verification of instrument translation (see section on National Adaptations 
and Translation Verification). 

• Verification of national data and marking procedures (see section on Data 
Management and Scaling). 

• Verification of population coverage and sampling requirements (see section on 
Target Population and Sampling). 

Eield procedures will be monitored by observers who are independent of the national 
centres and nominated by national research coordinators. It is envisaged to arrange 
interviews with natbnal coordinators in order to obtain information about how ICCS 
is being implemented and that a selection 15 selected schools be visited on their 
respective testing dates by quality monitors. During the school visit independent 




observers will record test administration, give a general impression of how procedures 
were followed and interview the school coordinator. 

A Manual for International Quality Control Monitors, elaborated in close cooperation 
between the lEA Secretariat and the ISC, will detail the envisaged monitoring 
activities and serve as a guideline for the independent observers. The lEA Secretariat 
will arrange the quality control monitoring in cooperation with International Study 
Centre and DEC and arrange an international training session for cjiality monitors 
from participating countries. 

After each data collection in the field trial and main study, NRCs will be requested to 
submit a standardised survey report describing how ICCS was implemented in each 
participating country. Together with the Test Administrator Session Report Eorms 
these data will provide an additional source for quality assurance. 

Data Management and Scaling 

Data collected in the ICCS study will be entered into data files with a common 
international format at the national centres using data entry software provided by the 
lEA Data Processing Center (DPC). National data files (together with all relevant 
information) will be submitted to the DPC for cleaning and verification. After 
checking and processing the national data the DPC will provide the International 
Study Centre (ISC) with summary statistics and upon feedback from national 
countries and the International Study Centre it will construct the international 
database. These procedures will be applied (at different levels of scope) for both field 
trial and main data collection. 

The reliability of marking procedures for open-ended test questions in participating 
countries will be monitored through the analysis of data derived from the multiple 
marking of random student sub- samples. These analyses give information on the 
degree of consistency across markers in each country and will be carried out by the 
International Study Centre in cooperation with the DPC. 

Sampling weights will be computed by the DPC in cooperation with the International 
Study Centre and added to the final international database. In order to allow the use of 
replication methods for the estimation of correct standard errors, it will also be 
necessary to provide the necessary information on sampling variance strata and 
replication indicators. It is envisaged that (similar to CIVED, TIMSS and FIRES) the 
jackknife repeated replication (JRR) be used for the estimation of standard errors in 
ICCS (see Schulz and Eehmann, 2004). 

The cognitive test data will be scaled using of IRT methodology (see Rasch, 1960; 
Hambleton and Swaminathan, 1991) and ability estimates will be plausible values in 
order to allow the estimation of measurement error and the estimation of latent 
correlations (see Mislevy, 1991; Adams, Wu and MacaskUl, 1997). 

Eurther scales will be derived from student background and student perception 
questionnaires. Student estimates for constructs could be derived as “Weighted 
Eikelihood Estimates” (WEE) (see Warm, 1989) using a partial credit IRT scaling 
model. 

The scaling review for ICCS items will comprise the following procedures: 

• Review of national and international item statistics provided by the lEA DPC 
(both field trial and main study). Item statistics should include classical item 




statistics (discrimination, point biserials, percentages) and IRT statistics (item 
parameter(s), item fit, item- by- country interaction). 

• Analysis of item dimensionality using Exploratory Factor Analysis, 
Confirmatory Factor Analysis and Multi-dimensional IRT scaling (both field 
trial and main study). Appropriate software should be used to deal with the 
problem of non- normality and the non- continuous nature of variables in 
covariance-based analyses. 

• Item selection (piloting, field trial) based on criteria developed regarding 
scaling properties and model fit. Here, a larger set of items will be scrutinised 
in order to select the most appropriate items for the main study. 

• Item adjudication (main study) both at the international (selection of final set 
of items) and national level (exclusion of national items from scaling in case 
of translation errors or misprints). 

Item calibration will be carried out based on an international calibration sample (500 
students per country, equivalent sub -samples of teacher and school data in case of 
scaling any items from these instruments). International item parameters derived from 
this calibration will then be used to scale the national datasets. Student (as well as 
potential teacher or school) estimates could be transformed to an international metric."^ 

For the reporting of trends it will be necessary to equate test scores from ICCS with 
those from CIVED. Equating would be facilitated through including all (or the largest 
possible subset) of the cognitive items used in CIVED (excepting those which were 
released in subsequent reports). A thorough analysis will be implemented in order to 
review the behaviour of link items comparing item parameters from both assessments 
and to ensure that only those items are selected for equating that have similar item 
difficulties. 

Data Analysis, Reporting and Dissemination 

The analysis of the field trial data will be coordinated by the International Study 
Centre (ISC) in close cooperation with the lEA Data Processing Centre (DPC) and 
guided by advice from the Project Advisory Committee (PAC) and the IE A 
Secretariat. It is envisaged that the analyses would be carried out by the International 
Study Centre in close cooperation with the DPC, expert consultants and staff working 
at the institutions associated with the International Study Centre and its partner 
institutions. 

It is deemed essential to elaborate a detailed draft analysis plan about half a year prior 
to the analysis phase in order to have sufficient time to discuss and organise the 
different analysis tasks between expert groups, lEA Secretariat and the International 
Study Centre and its partner institutions. The analysis plan needs to be guided by the 
assessment framework and the interests of countries participating in the study. 

It is proposed to use a wide range of statistical analyses including descriptive statistics 
(means, percentages, percentiles), tests of differences (between countries or subgroups 



In CIVED, the cognitive test scores had an international mean of 100 and a standard deviation of 20 
(see Schulz and Sibberns, 2004). It is currently proposed to change this metric to a different metric 
(with an international mean of 500 and a standard deviation of 100) in order to be consistent with the 
practice in major lEA studies like TIMSS and PIRLS. 




within countries), bi-variate analyses (effect coefficients), multivariate analyses 
(multiple regression) and multi-level analyses. 

The data analysis undertaken for reporting should: 

• address the research questions elaborated in the assessment framework; 

• be obtained by appropriate statistical methods; 

• be communicated and displayed in ways understood by potential readers; and 

• be reported accompanied by the methodological information about how the 
results were derived. 

In accordance with the lEA standards for international large-scale assessments, all 
population estimates will be reported with their respective standard errors. In view of 
the complex sampling design, replication methods would be used for estimating 
standard errors through the use of appropriate macros (SPSS or SAS) and/or specially 
designed statistical software (like WES VAR). 

It is viewed as very important for the success of this study to include a wide range of 
expertise in the review of the analysis outcomes. Drafts of all reports will be 
circulated among both experts and national centres, as appropriate. This includes the 
international report, contexts survey report and regional report (where appropriate). 
National centres will be invited to comment on the drafts and make suggestions for 
modifications or amendments, in particular with respect to the interpretations of 
national results and their relationship with specific country and/or regional contexts. 

It is envisaged that the following reports be published: 

• The International Report on the ICCS results will be published in 2010. This 
volume would include results derived using the international core instruments 
and be the equivalent to the first international report on CIVED (Tomey-Purta, 
Eehmann, Oswald and Schulz, 2001). 

• The Technical Report for ICCS will be published in 2010. This report would 
contain a description of details on instrument development, data collection 
procedures and analysis (see as an example the CIVED technical report in: 
Schulz and Sibbems, 2004). 

• Consideration will be given to publishing separate Regional Reports on 
regional components of ICCS. Eor example, there is a strong case for 
publishing a separate European Regional Report and Latin American Regional 
Report on the results derived using the regional instruments from the 
European regional module. 

National centres will elaborate national reports on the results of international studies 
and are often in need of assistance with analyses and reporting. Therefore, it is 
envisaged to provide workshops on data analysis and reporting after the data 
collection at the NRC meeting in November 2009. National centre staff will receive 
some basic information about the use of SPSS macros to appropriately analyse the 
ICCS data using a preliminary dataset (without any country results) as examples. 
Countries will also receive guidelines regarding the adequate reporting of results in 
natbnal reports. Using the expertise gained in CIVED and other international studies, 
the NRC meeting may also include discussions about how to ensure the results are 
disseminated effectively to a range of audiences, notably policy-makers, at national 




level. It is important that the efforts in coordinating and reporting the study at 
international level are matched by those at national and regional levels. 

After the release of the international database and the technical report, an 
International Database Seminar will be conducted in November 2010. At this seminar 
training for the analysis of the final ICCS data will be provided, including information 
on the basic statistical concepts, the correct use of statistical software and the 
interpretation of results. 

It is envisaged that reports and databases be made available through the Internet. In 
order to enable fellow researchers to use the ICCS data, the data files with the 
international database (from students, teachers and schools) and, where appropriate, 
regional database(s) would be made available for downloading at the ICCS website 
together with the necessary technical information (technical report, codebooks, 
instruments except the non-release test items, database description). National centres 
will also be encouraged to make their national reports available through the ICCS 
website. The web information will be elaborated in close cooperation between 
International Study Centre and its partner institutions, the lEA Data Processing Centre 
and the lEA Secretariat. 

Benefits for Participating Countries 

Benefits for countries that participate in the international part of ICCS will be the 
following: 

• National indicators of current knowledge and understandings in civics and 
citizenship among lower secondary students that may inform practices and 
policies in this area. 

• National indicators reflecting the beliefs, attitudes and behaviour of their 14- 
year- olds that will provide an informative picture of the civic and citizenship 
engagement of young people. 

• Comparative data that enable them to compare their national results with those 
from other countries. 

• In case of participation in CIVED 1999, the opportunity to assess changes in 
indicators of student knowledge and engagement over the past decade. 

• An assessment of the national status of civic and citizenship education from 
different perspectives including student learning outcomes, teacher 
perceptions and school- level practices. 

• Active participation in the development of the instrumentation of this study. 

• Opportunities for training in test development, operational procedures and data 
analysis. 

• The possibility of assessing country- specific issues administering additional 
national items at the end of international (and regional) instruments. 

Eurthermore, the option of participating in regional modules offers the following 
additional benefits: 

• Active involvement in the development of region- specific assessment of 
student knowledge, attitudes and engagement. 




• Region- specific indicators of student learning outcomes that may inform about 
areas not covered in the international instruments. 

• Opportunities to compare specific indicators with other countries from the 
same region. 
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